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[57] ABSTRACT 

To control congestion in packet switching networks, 
control of the traffic sent by a given station to each of 
the downstream nodes to which it is directly connected 
is efTected by control of the traffic that the upstream 
nodes to which it is directly connected are permitted to 
send to it. In this regard, a predictive model is used to 
predict the cross traffic, one round trip delay in advance 
that the given station can expect. TCe:parameteTs for 
the-predictive:mc^el:are:obuined:by:ro^ 
real;tim^a^d:by;the:uM;of:moving averages. Usingithe^ 
predicted:(*cra:traffic,-the^^ 
th^tl^proxim^ 

4^e^[om^^gWen^i^e, andrth^correcustaTe^rth"? 

8L v ^50^»Ztlw^sute:of;the:giVe 

d^y intcahe:futu^ This prediction is used 

to schedule the amount of traffic to be sent by each of its 

proximate upstream nodes. 

6 Claims, 2 Drawing Sheets 
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PREDirrrvF rnNrrircnnu rnvro at ™ ° f the ° rder of q u eueing delays (which is low at high 

HIGH-SPEED WIDE AREA NETWORKS tions, if a low cell loss can be guaranteed. Cell loss on 

iiMTPnni Tr-rinxr m other hand Ieads to retransmission, and the resulting 

INTRODUCTION 5 delay ^ ^ of the ordcr of propagation deIay Suc g 

This invention relates to packet switching networks congestion avoidance strategies are generally based on 
and more particularly to such network that includes controlling the rate at which cells are admitted into the 
provision for congestion control. Congestion control network. In most static rate control schemes, an esti- 
involves limiting the rate at which traffic is entered into mate of the average available bandwidth (for the class 
a network to a rate at which the network can transport 10 of traffic being controlled) is made. For example, if the 
the traffic to its intended destination without inordinate average bandwidth usage by higher priority services, 
dcla y* such as voice and video, is Ajr (this is known from the 

BACKGROUND OF THE INVENTION Cal1 admission control), and if the link bandwidth is X^, 

AJ . • . - ' , , then the average available excess capacity Xc=X/-Xr. 

Advances in high-speed packet switching and fiber 15 The static rate controls ensure that the combined arrival 
optic technology have opened the possibility of provid- rate of the rate controlled low priority class is limited to 
and transm^sion capacity from several some value rl c (0<r < 1). The problem with this ap- 
JSJ^S^ ^ GDits/second in a packet pr0 ach is that in the short run, the excess capacity can 
Tilt ™™°»™ nt ' " owever ' the Processing power momentarily reduce to zero and can remain at that leve" 
and the ability to provide large amount of high speed 20 for g time ^ . h , 

buffers may not keep pace with the advances in trans- availablc at the t * no<Je SK^taS to 

mission speeds. As a consequence, the dominant source i flrOA „mi w*o, ■ n -e\i / ■ , _ 
of packet^ cell loss in such networks would be due to ^^S^^^ ? f * Ct ° f r " ^P' f 0r 
buffer overflow. In low speed networks, flow and con- r w ^ r° ? than 
gestion control have been based on reactive mecha- 25 of the Imk. Several end. 

nisms. These include the logical link controls, and end- t !, ^VW** wmch att ^pt to 

to-end flow controls based on windows or credits. Such con «« tlon regulate the sources accordingly, 

reactive or feedback controls essentially throttle the End - to f nd adaptive ^schemes however can be effective 
upstream node or the source, as the case may be, when 0nIy whe " the round tn P P^pagation delays are small 
the network experiences congestion. Such controls 30 compared to the queue time constants. It can be shown 
have been effective because the queueing delays in low ™* tne ada P t!Ve end-to-end rate control scheme can at 
speed networks have been much larger than the propa- **** P erform onl y * well as an optimum static rate 
gation delays. As a result, the sources were able to react c °ntrol scheme. The disadvantage of the static rate 
to overloads before the state of the system changed control scheme is that if low cell loss is required, then 
significantly. In contrast, in high speed networks, where 35 ! ne **&emt results in low utilization. Any attempt to 
propagation delay dominates, reactive feedback con- increase the utilization results in an increase in cell loss, 
trols tend to be too slow to be effective. The speed with However, end-to-end control schemes may be effective 
which congestion sets in or dissipates is proportional to in ™8 n speed networks with small round trip delays 
the speed of the link. Because the state of individual sucft MANs and LANs. 

nodes can change rapidly, feedback information will be 40 Feedback flow control can be classified as static or 
out of date if the time to propagate the information is adaptive, and it can be based on an end-to-end window 
longer than the node time constants. mechanism or can be rate based. In the static window 

A common flow or congestion control scheme in low mechanism, the number of unacknowledged messages 
speed networks has been the use of end-to-end flow m tne network is limited by the window size. In adapt- 
control based on windows. For example, for each con- 45 ive window control schemes, the congested nodes send 
nection or virtual circuit (VC), the maximum number of congestion signals to the source and the source in turn 
unacknowledged packets is limited to a window of W responds by modifying its window size. In one known 
packets, In order to ensure very low packet loss due to system, the optimum end-to-end window is measured 
too many VCs becoming active at the same time, one by the round trip response time of packets and the win- 
has to allocate a window's worth of buffer for every VC 50 dow is adjusted dynamically. In another known system, 
at every node on the VCs path. In high speed wide area a feed forward rate control at the access node operates 
networks, the number of VCs can be very large, and the under an end-to-end window control that stabilizes the 
required window to obtain high throughput is also large open loop rate control. In the event of congestion, the 
because of the large bandwidth delay product, and this end-to-end window control can be reduced by the con- 
can translate to an unrealistic number of buffers in the 55 gested nodes which in turn quenches the source. The 
network. Further, a combination of large number of source responds by reducing its rate. In another system, 
VCs, each with a large window, can lead to severe a packet pair probing technique in conjunction with 
congestion when too many VCs become active at the round robin scheduling by virtual circuits at each node 
same time. An alternative is to use an adaptive end-to- is used to determine the level of congestion in the net- 
end control, where the window size is modulated based 60 work, and an appropriate control algorithm is used to 
on the state of the nodes on the VCs path. However, control the source rate. 

the effectiveness of such a scheme diminishes as the In another study, the performance of a threshold 
bandwidth delay product increases. based feedback control policy in a high-speed network 

Unlike reactive controls that allow congestion to with significant propagation delay has been investi- 
develop in the first place and then react to it, congestion 65 gated. The principal difficulty with high-speed wire 
avoidance strategies try to avoid congestion in the first area networks is the long round trip delay, which makes 
place, through conservative admission policies. Such end-to-end controls ineffective. A knowledge of the 
admission controls result in admission delays which are state of the nodes on the VCs path cannot be exploited 
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because of the large round trip propagation delays. transfer to the transmission channel for transmission 
Since the timeliness of the state information is deter- therealong its assigned destination, 
mined by the propagation delay and the rate at which In accordance with the preferred form of the inven- 
the state of the queues are changing, myopic controls tion, the cross traffic at each node is modelled by a first 
are more meaningful. For example, feedback controls 5 order autoregressive function with its parameters esti- 
between adjacent nodes that are not separated by large mated in real time, and this cross traffic model is used to 
delays will be more effective than feedback between predict the evolution of the queues of cross and con- 
edge nodes. trolled traffic at each node. 

■ SUMMARY np thf TWFNTTrtM , Basically, at the beginning of each time frame set for 

SUMMARY OF THE INVENTION l0 Ae systetnf wch nodc in thc nctwork nolcs hs own 

In the light of the above consideration, the present state > given by the content of its buffers and the amount 
invention uses hop-by-hop flow control, based on a of cr °ss traffic it received during the last frame. For 
policy that predicts the state of a given node in advance cacn ^ rarnc it also receives information from its down- 
by at least one round trip delay. In particular, a predic- stream neighbor; for example, node 14B will receive 
tive model is used to predict the cross traffic one round 15 information from node 14C about the maximum amount 
trip delay in advance. The parameters for the predictive of controlled traffic that node 14B can safely send it 
model are obtained by measurements in real time and by during the next frame, in light of the amount and type of 
the use of moving averages. Using the predicted cross traffic that nodc 14C expects to originate of its own. 
traffic, the amount of controlled traffic that the node Node 14B ! hcn » prediction as to what will be the 
downstream of the given node can accommodate from 20 content of its buffers in the next three frames in the light 
the given node, and the current state of the given node, of the amount of cross traffic it expects to receive and to 
the state of the given node one round trip delay into the transmit timely to node 14C. Based on the above infor- 
future is predicted. Knowledge of the future state of the ™t»n f node 14B tells it upstream neighbor, node 14A, 
given node is used to adjust the amount of controlled the maximum amount of controlled traffic that node 
traffic at the node upstream of the given node that will 25 ™ " n scnd t0 node l4B< ^ decision on the amount 
be sent down to the given nodc. As a consequence, the of f? 6 * UA Wl11 Permitted to send is based on 
given node generally receives only as much traffic as it ? ™™ between getting high throughput and avoid- 
can safely pass, so that congestion at such node is well In « b ™ cr ° vc ™ w Bt "ode 14B If the amount of traffic 
controlled ^ by node l4A 1S unne cessanly low, throughput may 

As the terms are used herein, "controlled traffic- is 30 ™ ncccs ? ^ ^.!*J hc probability of buffer 
traffic, such as data, that can tolerate appreciable delay ? VCrfl ° W a node * 4B *?" * low A Alternatively, if to 
and so is readily controlled, as contrasted with real time ™j™ ^ZJ^S^^ ^fr^rfS 1 *" * 
traffic, such as voice and video, known as -cross traf- "2? "J'J^ ^ of buffer over- 

lie." that cannot tolerate significant delay, and so needs , 5 fl ^fJ^ ■ 
.have priority in transmission and is not readily con- * on^n^ 

factor that make for prompt adjustment of traffic pat- 
BRIEF DESCRIPTION OF THE DRAWING tern changes. Additionally, it is receiver initiated, the 

speed packet switching transmission. m% makcs it J for fl d fc downstream £ ™ 

FIG. 2 is a chart showing the basic steps of the con- nected t0 a iun ^ of ^ no<Jcs 

trol process in accordance with the mvention. amount of ^ ^ ft is P wi]]i tQ receive ^ c 

FIG. 3 shows a model of a network that includes upstream node t0 ^ ^ it is * M] ^ . b ff 

provisions for congestion control m accordance with 45 capacity with , ittle risk of overflowmg its capadty 

the invention. There Mlom g mofe forma] descriptk £ of thc 

DETAILED DESCRIPTION OF THE modcl 

INVENTION Consider a tandem connection of K+2 nodes where 

r # .* . _ ljr * , . node 0 is the source node, 1 to K the intermediate nodes. 

With reference now to the drawing, FIG. 1 shows a 50 and node K+ 1 the destination of the controlled traffic! 

transmission network for interconnecting a transmitter We refer to such a network as a K node network We 

10 of signal information, or traffic, in packet form repre- as SUnic that nodc 0 has an infinite backlog of controlled 

sentative of voice, video and data signals and a receiver traffic. At intermediate nodes 1 through K we have 

12 of the signal information, the transmitter and re- also cross traffic, which has priority for service over the 

ceiver being interconnected by a suitable broadband 55 controlled traffic. The number of buffers at each nodc is 

transmission channel that includes a plurality of inter- f m j tc an d is shared by both traffic types. Time is divided 

mediate tandem nodes 14A, 14B, 14C, spaced along the into frames; the n<* frame begins at time U for n= 1 2 

channel at each of which traffic may be abstracted from . , Ut the propagation delay be T. between any two 

or added to the channel Nodes 14A and 14C will be adjacent nodes. We assume that each intermediate node 

described as upstream and downstream, respectively, of 60 measures the number of cross traffic cells that arrive 

node 14B. Typically, there will be large number of over successive non-overlapping intervals of length T M 

nodes in the network. We refer to these intervals as measurement intervals. 

It will be assumed that at each node, including those For the purpose of the discussion to follow, we assume 
corresponding to the transmitter and the receiver, con- that Ttf=T,-t. + i-u. Later, we will relax this as- 
trolled data traffic has to compete with video and voice 65 sumption. 

cross traffic for access so that buffers (not shown) are Let us define the following quantities: 

included at each node where input traffic in packet form X/(n)— Amount of cross traffic received by node j 

can be stored temporarily in queues while awaiting during frame n 
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Iy(n)— Amount of cross traffic transmitted by node j and the fraction attributed to the controlled traffic is 
during frame n 1 - p. From these statements, we have 

V>(n>— Maximum amount of controlled traffic that 
node j is allowed to transmit to node j+ 1 during ^<"+ #(«+ »)-/>[ 5W 0+ W(n+ \)~B max ). 

frame n. 5 

tf(n)-Actual amount of controlled traffic transmit- ^,+i)=^(n + i)-<i-^(n+iH Wan- 

ted by node j to mode j + 1 during frame n (7) 

U/00-Amoun. of cross traffic queued at node j, at tf ^+,,+^+1) >B m „. Since the cross traffic 

™,/,„x „„, - . ... . , „ JtJ in has Priority of service over the controlled traffic, the 

f .ll^? of rolled traffic queued at node >° 8moum of cros$ trafllc x „ ei by node j ^ frame ; h 
j» ai lime ifl. 

B*(n)— Total traffic queued at node j, at time t„ zy<")=rain{cW. <*o+Jifti>l (8 ) 

<U/(n)+W/(n)). 

CffiaV— Maximum amount of traffic that node j can Similarly, the amount of controlled traffic transmitted 

transmit in one frame. This is also the link capacity. 5 by node j during frame n is 
Bmax — Buffer capacity at each node. 

At time t ff -i, node j receives the decision V/(n— 1) ^(»)^ram{c m J-nf(n) r W(n)+p-\n-i) t ^{n)}. (9) 

from node This is the maximum amount of con- 
trolled traffic that node j is allowed to transmit to node - n ^h* f tts * term m - toe braces is the available capacity 
j-t- 1 in frame n-1. At time t„_i, node j also knows the during frame n after serving the high priority (cross) 
cross traffic that it has received during frame n— 2, traffic. The second is the total buffer content due to 
which is X^(n-2). Assuming that we know all the quan- controlled traffic and the third is the maximum amount 
tities listed above, we now describe the dynamics of the ?f controlled traffic that node j is allowed to send dur- 
system. For n= 1,2,3, ... we make the following as- 2 5 ing frame n * 

sumptions. In 0M9) above, we have shown how the quantities 

of interest are related to one another. In other words, if 

j%)= v°{n). (i) all the quantities were known, they would satisfy the 

equations (l)-(9). However, in the real problem, not all 

^ + w " 86 • (2) 30 quantities are known. In fact, there are five that we need 

to study. 

Here (1) implies that the source node (node 0) has an 1. What is an adequate model of XS(n) for n = 1 2 
infinite backlog of controlled traffic and (2) implies that and j= 1, . . . , K+ l?This will be the model for the 

the destination absorbs all controlled traffic. Before cross traffic. 

writing down the equations, let us define the two addi- 35 2. For the model of item 1, how does one estimate the 
tional items; parameters? This is the estimation problem. 

Fa tit v^tiv * rv/ * m 3 * Whal is a good wav t0 dete ™ne the decision 

w< " +,) - w+*w-"o* (3) variable v , (n)? This is the contro] problcm 

W( n+ i)=w(«) + r-i( fl -i)- J p(„). (4) 4 ' What » ■ reasonable way to predict the buffer 

40 content three frames from now? This is the predic- 

In (3), U/(n+l) refers to the amount of cross traffic < V? n P?°^ em * 

queued at node j at time W i under the additional as- * Ho " * thc to be modified if Tj**V 

sumption that there was no buffer overflow in frame n. We address each of these ,ssues m turn ' 
In (4), W/(n+ 1) refers to the amount of controlled traf- MODEL FOR CROSS TRAFFIC 

fie queued at node j at time t„+\ under the additional 45 n *«■■■•• j j J , J , 

assumption that there was no buffer overflow in frame . T? C CT ° s r s tra f^ ls intended t0 , model delay sensitive 
n. Clearly, the assumption that there was no buffer tra ^ c and for * hls * receives high priority. It is 

overflow in frame n is correct if Otn + I)+W*n+l) we " * n ° w V hat delay f enimve traff,c Iike video is 
ZBmax. From this, we have correlated. To capture this, we propose a first order 

50 autoregressive process to model the cross traffic. Thus, 
we have 

t*« + t) = &< n + i) U (f) + !)+ ^ (rt + l)S1? _ n.>-^.-i) + i*.-i) 

W{n + 1) ~ W(n + 1) J 

55 where Y/(n— 1) is a sequence of non-negative and inde- 
We now discuss the case when a buffer overflow oc- ^S^Z °* ^ cor T? ,ation 

curs, i.c, LV(n + l) + W/(n + l)>B m „. Since the cross ^S^L iSi ^^/T^i^^^ ^ P u" 
traffic as well as the controlled traffic compete for the lag k is given by (a,)* 

same buffer space, it seems reasonable to assume that „ Jt^'fi?!^, ^fL^^Tr' pr u°" 
their losses are proportional to the amount of traffic 60 f^Sf^L be used to define the 

received from them during frame n. Thus, the fraction p Iorc 8 om « process. 

of the total amount of overflow, attributed to cross THE ESTIMATION PROBLEM 

1C 1S Since the model of (10) is used in the prediction pro- 

65 cess (to follow in the subsection), the parameters of the 

p - % < 6) model are not known a priori. The estimation problem 

x!{n) + P-Hn - I) * essentially takes a moving window of J frames. At time 

t n , we estimate the mean of X/(n) by 



(10) 
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1. The prediction problem becomes easy to imple- 
. j n- 1 (u) ment as we do not have to keep track of the distri- 
K/fl) - 7* t^n-j buttons of various quantities. 

2. The assumption of linearity introduces bias in the 
We estimate the variance of X/(n) by 5 calculations. 

However, the bias is likely to be very small because 
n _ l both buffer overflow and empty buffers should be un- 

o-fa) = ±* {A7(o>2 - {£/n)) 2 . likely events for a well designed system. Our fundaraen- 

' "~ tal goal in this problem is to achieve low losses and high 

Finally, the estimated correlation coefficient at lag 1 % ° ^S^^^S^ °* ^ ? 8 *H the 

• vcn b * source node implies that a reasonable control scheme 

* should avoid an empty buffer with high probability. 

Thus, the bias in the prediction problem is quite small 
r . „_i -] < I2 > indeed. 

a» ~ j^y ^2_^)W - l) - (A/n)} 2 \{&/(n)}' 1 . 15 We now describe the three cycles of calculations. 

Note that the circumflex (A) is used throughout the 
By taking expectations of (10), we can find the estimate ^ tl>C corres P ondin 8 P^icted or esti- 



mated value. 
20 CYCLE 1 

At time t„_ i, node j carries out the following calcula- 



of the mean of Y>(n - 1) as 

W[«-l)-A/«Xl-a/fl)). (13) 

Hereinafter parameters for use in the first order au- tions for P rcdictin g the results at time t. 
toregressive process model for cross traffic arrived at 

by the preceding estimation process shall be described 25 - 0 ~ S/n - \)x>\n - 2) + V(n - 2) from (10-13) 

as the estimated parameters. " '[T^H^ ~ 0 <" * ,)} from < 8 > 

r {W = Win ~ I) + JP(n - 1) - &(n - 1) from (3) 

THE CONTROL PROBLEM i a (n " i } Z^rJ {n " 2 L n sumption n (U) 

P{n - 1) = min{Q max - Dt(n - 1), ^ f r0 m (9) 

At time t„-i, node j knows the values of X/(n-2), - , „. ft . - J) + - 2), ^(« - ))} 

IZ-^n-S), D/(n-2), I/(n-2), V/(n-l), U/(n- 1) and 30 mn) = W(n " " + - 2 > - - D from (4) 

W/(n-l). At this time, node j is required to find yj nttM t . # . ,. A . 

V/-i(n). This is the maximum amount of controlled Note < hat f m 14 > we assumm « tha ? l he actual 

traffic that node j-1 will be allowed to send to node j a ™ Unt f # cont ? 1Icd ^ffic transmitted I/-»(n-2) by 

in frame n. This is the decision variable and we provide • J " *' *?, n V ln / ram f. n - 2 15 1"* 1 * C( * ual t0 the 

a P-Rule in which the criterion is to predict the total ^ maximum ^ oy » cd Y a ^ ( "" 2) ThlS " the m0St 

buffer occupancy (U/(n+2)+W/(n+2)) at node j at ° bV, ° US P redlct,on for ^'(n-2) at time t..,. 

time n+ 2. This quantity is a function of the decision CYCLE 2 

variable V/-'(n). The P-Rule is to find the smallest a*#;«^» j • ^ ^ „ 

dieted value of the buffer occupancy three frames from *y _ - 

now to a value which is as close as possible to a certain 'l/T nfo,? n .x «/ , !!2 

fraction of the total available buffer. The parameter P - ? *<S" * " ^ 

tunes the control mechanism. If P is chosen to be very , + D = 0{n) + - Zy(n) f roro (3 ) 

large, one can expect large buffer overflows and if it is i~\ n - P - w~Hn - l) assumption II 

chosen to be too small, the throughput will be small. (n) = K(n " 0 (I6 ' 

THE PREDICTION PROBLEM _ . rH , ^ , ft . . . froro ( 9 ) 

The idea behind the prediction problem is to use the 5Q 

system dynamics described above. In using the system + ^ s ^ (rt) + ^"'t fl - i) - A«) from (4) 

dynamics, we need to make the following three assump- vt ^ . 

tions: Note that In ( 1 5), the estimate of V/(n - 1 ) is being set to 

L Not all the quantities above are known at time t„_ i . thc P rc . vious estimate. This is reasonable, since there is 

To circumvent this, we replace the unknown quan- 35 no addlt,onaI information available to update this esti- 

tities by their estimates as discussed above. mate * Sim,larlv > ln (16), the prediction of the maximum 

II. We assume that the actual amount of controlled 8mount pf controlled traffic V/(n) that node j can send 
traffic transmitted by node j-l in any frame !° J + 1 m frame n is sim P ] y c^al to the previous 
(IMC)) is exactly equal to the maximum value ^ own valu c as there is no additional information avail- 
allowed for that frame (V/- »(.)). ^ able t0 make a prediction. 

III. The system dynamics contain some non-linear CYCLE 3 
operations since the buffer content cannot fall A#+ . . . 

below 0 and cannot exceed B m „. So, it is hard to . time '"T 1, J CameS out the folIowin « calcula- 

take expectations of non-linear functions without * 0nS to prcd,ct thc rcsults at time 

having a complete knowledge of the distribution. 65 

To get around this difficulty, we ignore the non- B fl) " ^< rt ~ ') 

linear effects in the prediction problem. V" t ! > ~ ¥ n) 3 n) +-/( fl > , t ft . ^ r ™ Cio) 

Item III above has two implications: *** + 0 = mm( ^ " {n + 0 + Mn + 0) from (8) 
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•continued informs the upstream node of the maximum rate for the 

from (3) composite stream. The upstream node can now allocate 
u>(* 4 2)= u(n + i) + Ji7(» + i) - + i) the transmission capacity to the different flows on a first 

£i0i + i) » come first oasis t or it can allocate based on the 

J~Hn) m PB max - to\n + 2) - + i) + a b 5 backlog of cells at that node due to each stream. When 

min <<4» - + D. + D) there are multiple upstream nodes, in the absence of any 

knowledge of the number of VPs carrying data traffic 
f o if iP(n + 2) - PBmax * o traversing through a node, (i.e., the node assumes that 

\ j u \ in therc is on,y a VP or controlled flow), the down- 

V, * <■) otherwise 10 strcam nCK j c determines the rate V for the controlled 

traffic and informs all upstream nodes contributing to 
While the calculations in cycle 3 follow the usual pat- the controlled traffic. In this case, the combined arrival 
tern, Eq. (17) does not. We can prove that this choice of rate at the node in question can be MV, where M is the 
VM(n) actually satisfies the P-Rule specified above. total number of upstream nodes. This will lead to large 

MEASUREMENT INTERVAL AND UPDATE l ? ss * M wcl1 w oscillation s in the network. Alterna- 

INTERVALS tively, the down stream node can ask each of the M 

7 , , , , upstream nodes to limit the maximum rate of their con- 

.n™, 1 ^^* W x Z C aSSUmcd th8t thc mea " trolIed traffic t0 V/M until thc next update. While this 

ssats ? a t r • one unit °: 

in the interval T„the cross traffic rate does not change ^^^^j^^ ^ lm}tmg * w 
significantly 25 tra *" c ratc t0 V /M). A way to overcome this problem 

Consider the case when the measurement interval jt^K^f t0 exchan S e frequently, information about 

Tjvis less than the propagation delay T„ say T p =mT M ™? backI °* ,s characteristic of the preferred em- 

(m> 1). In this case, one can determined control rate bo *™« °f the mvennon 

in one of four ways. ^ nodcs *»■•<» M be the immediate upstream nodes 

1. The control rate information V/(n) is determined 30 ?J node M+l. Node i(i-l,2 M) informs node 

only once every m measurement periods, i,e., a ^ + J of 1 ^ backlog of controlled traffic W> and the rate 

control signal is generated once every Tp=mTjtf. ?' a ! which u has **** receiving controlled traffic from 

In this case, we refer to T p as the update period. lts Sl "niediate upstream nodes, as well as the available 

Note that in this case, the prediction of the cross transmission capacity (C'-DO for controlled traffic on 

traffic and the evolution of the queue are carried 3 * ^ link connecting node i arid node M + l, where C'is 

over (3m + 1) cycles. The computational over-head tnc transmission capacity of the link connecting node i 

for each control signal is 0(3m). However, only a and node M + and D'is the capacity consumed by the 

single rate V/(n) is established for each update per- cross traffic on this link. Note that the controlled traffic 

iod. we are referring to at node i, is the controlled traffic 

2. Alternatively, for every update update period, one 40 component that will be routed from node i to node 
can determine a rate vector with m elements, one M + 1. Based on this information, node M+ 1 allocates a 
for each of the m sub-intervals. This control vector rate P/V to node i, where 

is generated once every Tp, but the upstream node 

modifies its rate at the beginning of each sub-inter- . , . ^ 

val. For this method, the computational cost for 45 /> = " mlc ~ fft w + P) — . 

each update interval is 0(3m 2 ). 2 mm{cf - jy, Hi + p) 

3. One can also send a control signal at the end of J=s 1 
each measurement. This would result in m control _ 

signals every Tp While the computational cost is . notatlonal convenience, we have dropped the index 
0(3m*), the communication cost is m times the first 50 n m the above equation. Here we assume that the trans- 
method, mission rate that is allocated to the controlled traffic 

4. Finally, one can reduce the measurement and com- flow must be tne minimum of the available capacity 
putational overhead by making fewer measure- (C'-D') for control led traffic, and the controlled traffic 
ments, say once every Tm ~mT p (m> = 1). We call backlog (W'+IO. which is the maximum rate at which 
this method the aggregation method. 55 n0<,e * W *N be able to transmit its backlog of controlled 

Numerical results show that the first method is more traffic if node M+ 1 does not place any constraint on it 
than adequate. The method has low computational and Sucn * two way exchange of state information can 
communication overheads. While method 2 and 3 have significantly improve the efficiency of the control, espe- 
significant computational and communication cost, they cially when the loading is asymmetric, 
do not typically result in any significant improvement in 60 FIG. 2 illustrates the basic steps of the control pro- 
performance. Choice of an appropriate measurement cess carried out at each node to motivate and control 
interval is also discussed. congestion in accordance with the preferred mode of 

In the model under consideration we have assumed a the present invention, as discussed above with mathe- 

single controlled flow of data traffic in a tandem net- matical detail 

work. When there arc multiple flows of controlled traf- 65 The process will be described now with reference to 

fie, such as multiple virtual paths sharing the capacity of a particular node, termed the current node, each of the 

a common virtual pipe, each downstream node treats upstream nodes supplying the current node directly, 

the multiple flows like a single composite flow and termed thc proximate upstream nodes, and each of the 
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downstream nodes supplied directly by the current 
node, termed the proximate downstream nodes. The 
number of proximate upstream and downstream nodes 
may vary widely and be as few as one and as many as 
tens. 
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instant node by its upstream proximate nodes for use in 
step 110, Information provided at step 1U as to the fill 
of controlled traffic and cross traffic in the buffer asso- 
ciated with the current node and information as to the 
capacity of the downstream nodes to accept controlled 
traffic are used to schedule the cross and controlled 
traffic to transmit to the downstream nodes. 

FIG. 3 illustrates in schematic form three tandem 
nodes in a transmission path using congestion control in 
accordance with the invention. 

The three nodes are, node j-1, the upstream node, 
node j, the current node, and node j + 1, the down- 
stream node. 

Each node includes a receiver 20A, 20B, and 20C 



Eirst^there is continuous measurelnel^ imiicatea^as 
step 10lKat;the7current node, as-weIl as each other node, 
°fJh^rois:traffic:passin^ 
time-framcr^hicjHs^ 
the r ave£aje^pfopagatio 
nodes alongthe Hrans m^ 
injtoate^y:stej>^ra^ 

continu ously t he ave rag e cross tr affic JoadsTbvcZa wii£ 

dow ^spanning sevjgal-t^^ ^ u JIWC mtJUUCS a reviver xua, ana zuc 
are-useti-t o : rgeaMcjr^ 15 labeled Rx that receives the signal information coming 
autoregressivefuncticji^ into the node from the upstream proximate nodes, and a 
tp^^djed^nheimstant node at -time-frames (n+1), transmitter 22A, 22B, and 22C, labeled Tx that trans- 
("+2)rwi>ere^ mits signal information to its downstream proximate 
steps lOj^O^andllO^ nodes. In the interest of simplicity, only a single contin- 
As mojcated-by-s^ep^lOe^there ls^lso^ejngjnwsured 20 uous path has been shown, although at each receiver 
cominuouslyrat:thexurrent;n«Je:the.cont plural inputs, and at each transmitter plural outputs, 
atnyjngfr^ ^ typically will be involved. Moreover, at each input the 
thisiiherej^^ traffic will include both cross traffic and controlled 
Uhat:can„be expected to be available in frames (n+l)and> traffic which will be separated therein for individual 
r*'^*"^ 1 * 111416 !* 107 ^andrlOfc 25 treatment, the cross traffic being passed through whil- 
st Additionally, as indicated at step 110, there is re- out delay, the controlled traffic typically passing into a 
ceived continuously from each of the proximate down- buffer to be controllably measured. The metering is 
stream nodes information relating to its capacity to controlled by the controls 24A, 24B, and 24C at each 
accept traffic from the current node in the next time node. As previously discussed, this control is supplied 
frame (n + 1) as well as to estimate its capacity to accept 30 with information provided by the traffic monitors 26A 

Clirfl traffic in *im*» framec 1\ n*A /- i 1\ I-j: t >%rr* _ « ■ ' 



such traffic in time frames (n-f 2) and (n+3), as indi 
cated in steps 111 and 1 12. 

As indicated by step 114, there is continuously being 
monitored the degree of fill of controlled traffic in the 
buffer associated with the current node. This fill consti- 
tutes the queue of the node. Based on the queue fill for 
frame n, and the estimates provided by steps 107 and 
108 and the predictions of steps 103, 104, and 105, there 
is predicted the queue fill for frames (n + 1) (n + 2) and 
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26B, and 26C, which monitor the traffic that has come 
into its associated transmitter, information supplied by 
the control of each of its downstream proximate nodes, 
and information as to amount of controlled traffic in its 
queue. Eaph control in turn provides control informa- 
tion to schedulers 28A, 28B and 28C, which schedule 
the amount of controlled traffic to be passed from the 
queue into the associated transmitter. Each control 
additionally passes information into the control of each 



• * j- j . -uuiiiuiioiij f«ws" luiumiduun iniu me control oi eacn 

(n+3) as indicated by steps 115, 116, and 117 based on 40 of its proximate upstream nodes needed by such node 

the predicted cross and cnntrn11*»H traffic an<4 tl,* mi r n * 



the predicted cross and controlled traffics and the queue 
at the preceding time frame. Step 117 predicts only on 
the basis of the cross traffic expected at frame (n + 3) 
and the queue predicted for time frame (n+2). 



for its role. 

It is to be appreciated that this processing is essen- 
tially done instantaneously at each node by a properly 
programmed computer. Moreover, extensive numerical 



» , - t \- ■ ptvsnMuiiivu vviij}/ulgi. muicuvcr, extensive numenca 

Based on the various estimated queue fills provided 45 calculations on the expected performance of the hoD 
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by steps 115, 116 and 117, as step 118, there is estimated 
the rate of controlled traffic that can be permitted to 
proceed from the upstream proximate nodes to the node 
in time frame (n+2). 

Additionally, as step 119, from the upstream node 
there is being supplied the backlogs in controlled traffic 
at each of the upstream proximate nodes and the avail- 
able link capacity between such nodes and the current 
node. Based on this information and the total controlled 
traffic that can be supplied to the downstream proxi- 
mate nodes available from step 118, as step 120 there is 
estimated the rate of controlled traffic that can be sup- 
plied from each upstream proximate node, and as step 
121 this information is sent to the upstream proximate 
nodes so each can transmit appropriately. 

There is also sent, as indicated by step 122, to the 
downstream proximate nodes its queue backlog and 
channel capacity information needed by each such node 
to control the traffic the current node will be allowed to 
send to the downstream node, in the manner that served 65 
to control the traffic the upstream proximate nodes 
were allowed to transmit to the current node. This 
essentially corresponds to the information sent to the 



60 



by-hop system described indicates that its expected 
performance compares favorably with the expected 
performance of alternative possible control systems, as 
discussed below. 

1, In comparison to both the static rate control policy 
(where the available capacity is known a priori 
from the call admission control) and the end-to-end 
based rate control policy, the predictive control 
policy performs significantly better in all cases that 
we have considered. For a given end-to-end cell 
loss rate for the controlled traffic, the predictive 
control policy is able to achieve higher network 
throughput. The actual improvement depends on 
the nature of the cross traffic. In the examples that 
we have investigated, the improvement in network 
throughput has ranged from 8% to 116%, and the 
improvement in controlled traffic throughput has 
ranged from 20% to 1200%. 

2. The cross traffic characteristics have a significant 
impact on the network performance. In particular, 
the performance degrades for larger variances and 
large correlation in the cross traffic arrival rate. 
However, the relative improvement in perfor- 
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mancc of the predictive policy when compared to 
the static rate control policy is more for higher 
values of the correlation coefficient and the 
squared coefficient of variation. This is particularly 
important as the high priority cross traffic, such as 5 
video, is likely to have correlations. In order to 
achieve low loss and high network throughput, the 
cross traffic load must be limited when the variance 
of its arrival rate is high. 

3. Like all reactive control mechanisms, the effective- 1° 
ness of the predictive control policy diminishes 
with increasing link speed and propagation delay. 
The actual degradation in performance depends on 
how the cross traffic load is scaled up with increase 

in link speeds (in order to keep the utilization con- 15 
stant). The degradation is much less when the cross 
traffic is due to the superposition of many low 
intensity streams. The efficiency of the predictive 
control policy at higher link speeds can however 
be improved by providing more buffers at each 20 
node. In the case of large propagation delay be- 
tween adjacent nodes, we propose that the links be 
segmented and connected through pseudo-nodes. 
The effect of introducing the pseudo-nodes is to 
make it appear as though the two nodes are closer 
than they really are. 

4. In the case of multiple data streams (which share a 
common virtual pipe at the downstream node), 
oscillations can result which leads to higher cell 30 
loss. If the upstream nodes inform the downstream 
node of their backlog of controlled traffic cells and 
the transmission capacity to transmit them, wast- 
age of network resources and cell loss rate can be 
reduced. 

5. A drawback with all feedback control policies is 
that sources that are near to the common feedback 
point receive preferential treatment. This favorable 
bias to the nearby sources can be eliminated by 
making the distant sources appear closer through ^ 
the introduction of pseudo-nodes. 

What is claimed is: 

1. A process for the control of traffic for reducing 
congestion on a transmission path that includes a plural- 
ity of nodes between an upstream source of the traffic 45 
and a downstream destination for the traffic comprising 
the steps at a given node of: 
predicting, on the basis of recently measured cross 
traffic, the amounts of cross traffic that it will re- 
ceive over each of the next three time frames; 50 
estimating, on the basis of measured controlled traffic 
being received from each upstream node to which 
it is directly connected, the amounts of controlled 
traffic that will be available to it over the next two 
time frames; 55 
based on (1) the said previously predicted amounts of 
cross traffic it will be receiving over the next three 
time frames, (2) said previously estimated amounts 
of controlled traffic that will be available to it from 
said upstream nodes over the next two time frames, 60 
(3) the estimated capacity to accept controlled 
traffic from it of the. downstream nodes directly 
connected to it, and (4) the present state of its 
queue, predicting the amounts of traffic that will 
keep its queue at a desired level over the next three 65 
time frames; 

based on the predictions of said last recited step, 
estimating the total amount of traffic it should ac- 
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cept from said upstream nodes two time frames 
later; 

based on the estimates of the last step, the controlled 
traffic backlog at each of said upstream nodes and 
the available link capacity of each, scheduling the 
rate of controlled traffic it is willing to accept from 
each of said upstream nodes; 

forwarding the information as to the scheduled rates 
to each of said upstream nodes; 

receiving from each of the downstream nodes di- 
rectly connected to it the information as to the 
scheduled rate of controlled traffic it can send to 
each of said downstream nodes; and 

transmitting its cross traffic and controlled traffic to 
each of said downstream nodes in accordance with 
the information received from each of the said 
downstream nodes. 

2. The process of claim 1 in which the prediction of 
the amount of cross traffic that the given node will 
receive over the next three time frames uses the average 
of the cross traffic over a window of several time 
frames. 

3. The process of claim 2 in which the traffic is in the 
form of signal packets and the cross traffic is primarily 
voice and video traffic and the controlled traffic is pri- 
marily data traffic. 

4. The process of claim 3 in which the time frame 
approximates the average time it takes the traffic to 
travel between adjacent nodes of the transmission path. 

5. In a packet switching transmission system includ- 
ing a plurality of nodes that are separated an average 
propagation delay time and where traffic may be added 
or subtracted from the path, a process for hop-by-hop 
congestion control that comprises at each node the steps 
of 

predicting the queue size at said node three propaga- 
tion delay times in advance, 

estimating the amount of traffic it can accept from its 
proximate upstream nodes two propagation delay 
times in advance based on such prediction, 

based on these estimates and predictions, transmitting 
to its proximate upstream nodes the amount of 
traffic it can accept for controlling the rate at 
which each transmits traffic to it, 

receiving from its proximate downstream nodes the 
traffic they are willing to accept from it, and 

transmitting to its proximate downstream nodes the 
amount of traffic each can accept from it. 

6. A packet switching transmission network that in- 
cludes a plurality of nodes separated by an average 
delay time and interconnected to provide hop-by-hop 
transmission flow between sources and destinations 
characterized in that hop-by-hop control of the flow is 
provided at each node by control means comprising 

means for predicting the queue size at said node three 
delay times in advance and for estimating the maxi- 
mum amount of traffic it can accordingly accept 
from its proximate upstream nodes two delay times 
in advance, 

means for transmitting these estimates of amount of 
traffic it can expect to each of its proximate up- 
stream nodes, 

means for receiving from its proximate downstream 
nodes similar information as to the amount of traf- 
fic that each will accept from it, and 

means for scheduling and transmitting to each of its 
downstream nodes traffic in accordance with the 
information received of the amount of traffic each 
can accept. 



10/29/2003, EAST version: 1.4.1 



